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0.0  Abstract 

In  Phase  I,  we  investigated  a  case-based  reasoning  (CBR)  approach  to  Operator 
Assessment  and  Operator  Machine  Interface  Enhancement  for  the  LAMPS  SH-60R  Multi 
Mission  Helicopter  Upgrade  (MMHU).  We  developed  a  limited  prototype  case-based  Operator 
Assessment  and  Operator  Machine  Interface  Enhancement  System  (OA/OMIES),  for  the  SH-60R 
sensor  operator  for  a  small  subset  of  ASW  situations.  We  developed  a  generic  OA/OMIES 
architecture  applicable  in  many  other  domains.  The  OA/OMIES  tests  operator  knowledge 
through  the  use  of  tactical  scenarios,  derives  the  operator’s  mental  model  based  on  his 
performance  and  explanations  for  his  actions,  then  adapts  the  operator  interface  based  on  his 
deficiencies  revealed  in  the  mental  model.  The  prototype  implementation  provided  an  absolute 
proof  by  example  of  the  feasibility  of  our  ideas.  The  case-based  approach  offers  the  further 
benefits  of  automatically  or  semi-automatically  generating  the  operator's  mental  model  and  of 
largely  circumventing  the  difficult  and  time-consuming  process  of  constructing  an  explicit  expert 
mental  model.  Our  approach  could  be  easily  extended  to  constitute  an  Intelligent  Tutoring 
System  (ITS)  for  the  SH-60R  as  well. 
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1.0  Executive  Summary 

The  Light  Airborne  Multi-Purpose  System  (LAMPS)  includes  several  tactical  sensors  and 
associated  signal  processing  and  display  equipment.  Each  has  different  modes,  settings,  and 
methods  of  operation.  Especially  in  the  case  of  the  radar  and  sonar  systems,  essential  to  optimal 
use  of  the  equipment  is  an  understanding  of  several  factors.  These  include  the  current 
environment  and  its  effects  on  the  signal  propagation  paths;  the  physics  of  the  signal 
propagation;  enemy  tactical  behavior  and  signal  source  characteristics;  and  the  capabilities, 
limitations  and  processing  algorithms  of  the  sensors  and  processing  systems.  The  sensor 
operator  (SENSO)  and  Air  Tactical  Officer  (ATO)  need  to  make  good  tactical  and  sensor  choices 
to  accurately  separate  enemies  from  non-enemies  and  perform  their  mission  well. 

Unfortunately,  because  this  is  such  a  complicated  domain  (or  actually  several  domains, 
since  there  are  several  mission  and  sensor  types),  the  knowledge  and  performance  of  ATOs  and 
SENSOs  varies  considerably.  One  solution  would  be  to  make  the  use  of  the  equipment  more 
automatic  by  automatically  selecting  sensor  operating  modes,  parameters,  and 
employment/deployment  methods.  However,  this  would  detract  from  the  flexibility  and 
therefore  the  capability  of  the  equipment  in  the  hands  of  the  most  expert  users.  What  is  needed  is 
a  system  which  can  assess  the  proficiency  and  knowledge  of  an  operator  in  the  various  types  of 
sensors  and  tactics  and  related  principles  of  operations,  assess  his  qualifications,  and  adjust  the 
equipment  appropriately. 

The  complex  set  of  principles,  which  is  required  knowledge  for  the  SENSO  and  ATO,  is 
duplicated  for  each  type  of  sensor,  tactic  and  mission  for  which  the  SENSO  and  ATO  are 
responsible.  The  sensors  include  radar,  EMS,  active  and  passive  sonar,  SAR,  ISAR,  and 
Magnetic  Anomaly  Detection  (MAD).  This  complexity  leads  to  a  complex  definition  of  operator 
proficiency.  Simple  linear  labels  such  as  Novice  through  Expert  trivialize  a  very  complex 
domain.  Instead  of  a  one-dimensional  description,  a  very  large  number  of  dimensions  is  required 
-  one  for  each  principle  or  related  set  of  principles,  which  implies  hundreds  of  dimensions.  A 
better  description  is  to  keep  track  of  the  set  of  principles  that  the  SENSO  or  ATO  has  current 
mastery  of  and  the  set  of  ones  on  which  he  is  weak. 

Simple  tests  of  what  principles  he  knows  and  doesn’t  know  are  not  sufficient.  Simply 
questioning  the  operator  on  the  principles  (with  multiple  choice  answers  for  example)  is  not 
sufficient,  since  what  is  most  important  is  how  the  principles  should  be  applied  in  a  tactical 
scenario. 

All  Phase  I  objectives  described  in  the  Phase  I  Proposal  were  accomplished.  In  Phase  I, 
we  investigated  a  case-based  reasoning  approach  to  intelligent  Operator  Assessment  and 
Operator  Machine  Interface  Enhancement  Systems  (OA/OMIESs).  We  developed  a  prototype 
case-based  OA/OMIES  within  the  LAMPS  SH-60R  MMHU  ASW  domain.  We  determined  the 
requirements,  both  hardware  and  software,  for  integrating  the  OA/OMIES  with  existing  systems. 

The  ultimate  Operator  Assessment/OMI  Enhancement  System  (OA/OMIES)  will  assess 
an  operator’s  knowledge  and  tactical  proficiency  by  testing  him  with  example  tactical  scenarios, 
off-line.  An  example  consists  of  a  problem  description,  solution,  and  explanation  or  steps 
leading  to  the  solution.  An  exercise  is  extracted  from  an  example  by  only  showing  the  operator 
the  problem.  He  must  then  generate  the  solution  himself.  His  solution  and  solution  steps  can  be 
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compared  to  that  of  the  exercise  for  grading,  deficiency  diagnosis,  and  interface  alteration.  The 
system  works  interactively  with  the  operator  to  test  his  knowledge  by  using  scenarios  of  sensor 
employment  in  tactical  situations.  These  scenarios  are  generally  presented  through  a  tactical 
simulation. 

In  order  to  tailor  the  equipment  operation  to  the  individual  operator,  the  system  will  keep 
a  model  of  each  operator  tested  using  the  OA/OMIES.  The  operator  model  will  contain  the 
operators'  actions  and  decisions  during  different  exercises,  the  principles,  procedures,  and 
techniques  which  have  been  tested,  and  those  that  have  been  mastered  based  on  performance  on 
exercises.  The  set  of  principles,  procedures,  rules,  and  tools  referenced  in  the  solutions  of 
problems  the  operator  has  solved  successfully  represent  the  operator’s  mastered  skills.  Based  on 
the  pattern  of  his  unsatisfactory  performance  on  exercises,  a  set  of  topics  and  principles,  or 
combinations  of  them,  can  be  developed  which  form  a  hypothesis  as  to  what  knowledge  the 
operator  does  not  understand.  This  hypothesis  is  the  basis  for  the  operator  model  which  will  be 
used  to  enhance  the  user  interface  to  counteract  his  deficiencies. 

The  OA/OMIES  will  then  make  use  of  the  operator  model  to  enhance  the  user  interface, 
in  a  way  which  is  customized  to  the  particular  operator  and  which  optimizes  the  combined 
operator/sensor  system  performance.  This  enhancement  may  include  automatically  setting 
sensor  operation  or  processing  modes,  parameters,  options,  etc.;  priming  certain  help  files  or 
features  for  the  operator;  recommend  certain  configuration  settings;  starting  and  initializing 
decision  aids  for  the  operator;  or  making  use  of  expert  systems  to  configure  the  equipment 
appropriately.  The  enhancement  can  be  performed  in  a  number  of  ways,  all  of  which  were 
investigated  and  possibly  will  be  implemented  in  parallel.  Since  the  operator  model  includes  the 
principles  and  skills  in  which  the  operator  is  weak,  these  are  passed  to  the  on-board  enhancement 
system  to  set  the  OMI  appropriately  for  the  given  circumstances. 

The  visualized  sequence  whereby  the  OMI  Adaptation  software  will  be  utilized  consists 
of  3  primary  phases.  The  first  is  evaluation.  In  the  evaluation  phase,  the  system  tests  operator 
knowledge  and  builds  a  model  of  his  knowledge.  Much  of  this  would  occur  by  testing  him  with 
a  tactical  simulation  and  analyzing  his  responses.  In  the  second  stage,  the  OMI  Adaptation 
system  would  present  proactively,  minimal  tailored  information  on  the  auxiliary  display  during 
an  actual  mission.  This  mission  might  be  either  a  training  mission  or  actual  combat.  Finally, 
especially  if  a  training  mission  was  performed,  the  third  phase  could  consist  of  a  debriefing. 

The  Phase  I  prototype  provides  absolute  proof  of  the  feasibility  of  our  ideas.  The  Phase  I 
prototype  implements  all  phases  of  the  full-scale  OA/OMIES,  though  on  a  very  narrow  part  of 
the  SH-60R  domain.  It  includes  both  an  assessment  module  which  tests  operator  knowledge  in 
scenarios  running  in  a  tactical  simulation,  and,  an  enhancement  module.  The  assessment  module 
assembles  an  operator  model,  consisting  of  what  Principles  the  operator  is  weak  and  strong  in. 
The  assessment  also  performs  assessment  efficiently.  That  is,  if  it  determines  that  he  knows  very 
little  or  very  much  about  one  part  of  the  SH-60R  domain,  it  marks  the  entire  area  accordingly  and 
moves  on  to  scenarios  covering  other  areas. 

The  enhancement  system  uses  the  operator  model,  in  the  context  of  the  current  situation, 
to  provide  the  appropriate  enhancements.  Enhancements  included  in  the  Phase  I  prototype 
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include  recreation/improvement  of  existing  sensor  displays,  knowledge-based  advice,  advisories, 
warnings,  suggestions,  explanations,  and  domain  information  (both  general  and  tailored  to  the 
context).  Section  6  lists  the  prototypes  capabilities  in  more  detail  and  Appendix  A  gives  a  series 
of  screen  dumps  which  illustrate  a  demonstration  sequence  which  clearly  shows  the  feasibility  of 
our  concepts. 

The  ultimate  goal  of  this  project  is  a  fielded,  operational  system  which  performs  off-line 
assessment  and  on-line  OMI  enhancement,  on-board  the  SH-60R,  for  both  the  ATO  and  SO 
positions.  This  is  an  enormous  scope  which  must  be  scaled-back  and  prioritized  for  Phase  II.  In 
Phase  II,  we  would  produce  an  operational  prototype,  ready  for  testing  and  evaluation,  probably 
interfaced  through  an  RS-232  port  to  a  land-based  functional  cockpit  mock-up.  The  Phase  II 
system  would  handle  a  subset  of  the  applicable  knowledge  and  tasks  of  the  SO  or  ATO.  The 
ultimate  system,  in  addition  to  interfacing  to  the  actual  SH-60R  avionics  must  also  interface  to  an 
SH-60R  trainer,  for  OA/OMIES  testing,  off-line  assessment,  and  for  training  the  crew  in  the  use 
of  the  on-line  enhancements. 

Future  work  will  include  both  the  development  of  applicable  OMI  enhancements  by 
SHAI  as  well  as  incorporation  of  enhancements  developed  by  others.  The  Decision  Support 
System  (DSS)  is  one  example.  Our  architecture  minimizes  the  difficulty  of  incorporating 
enhancements  developed  by  other  organizations.  SHAI  is  qualified  to  develop  several  different 
types  of  enhancements.  Which  ones  we  develop,  and  which  ones  will  be  developed  by  others, 
must  be  decided. 
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2.0  Background 
2.1  SH-60R 

The  Light  Airborne  Multi-Purpose  System  (LAMPS)  includes  several  tactical  sensors  and 
associated  signal  processing  and  display  equipment.  The  sensors  include  a  dipping  hydrophone, 
passive  and  active  sonobuoys,  EMS,  radar,  and  Magnetic  Anomaly  Detection  (MAD).  Each  has 
different  modes,  settings,  and  methods  of  operation.  Especially  in  the  case  of  the  radar  and  sonar 
systems,  essential  to  optimal  use  of  the  equipment  is  an  understanding  of  several  factors.  These 
include  the  current  environment  and  its  effects  on  the  signal  propagation  paths;  the  physics  of  the 
signal  propagation;  enemy  tactical  behavior  and  signal  source  characteristics;  and  the 
capabilities,  limitations  and  processing  algorithms  of  the  sensors  and  processing  systems.  This  is 
especially  true  in  littoral  environments  where  shallow  water,  near  or  over- flown  land  masses,  and 
a  large  number  of  commercial  and  neutral  surface  and  airborne  contacts  significantly  complicates 
the  sensor  optimization  problem.  Because  of  the  clutter  and  multipath  effects  in  littoral 
environments,  the  sensor  operator  (SENSO)  needs  to  make  good  sensor  choices  to  accurately 
separate  enemies  from  non-enemies. 

The  problem  is  further  complicated  by  the  tactical  ramifications  of  the  sensors.  Use  of 
active  sensors  often  reveals  the  presence  of  the  LAMPS.  This  loss  of  the  surprise  is  serious 
enough  in  its  Antisubmarine  Warfare  (ASW)  mission,  providing  warning  to  a  submarine  that  it 
has  been  detected,  but  it  is  far  worse  in  its  Anti-Ship  Surveillance  and  Targeting  (ASST)  mission, 
perhaps  making  the  helicopter  a  target  of  attack,  itself. 

Unfortunately,  because  this  is  such  a  complicated  domain  (or  actually  several  domains, 
since  there  are  several  sensor  types),  the  knowledge  and  performance  of  SENSOs  varies 
considerably.  One  solution  would  be  to  make  the  use  of  the  equipment  more  automatic  by 
automatically  selecting  sensor  operating  modes,  parameters,  and  employment/deployment 
methods.  However,  this  would  detract  from  the  flexibility  and  therefore  the  capability  of  the 
equipment  in  the  hands  of  the  most  expert  users.  What  is  needed  is  a  system  which  can  assess 
the  proficiency  and  knowledge  of  an  operator  in  the  various  types  of  sensors  and  related 
principles  of  operations,  assess  his  qualifications,  and  adjust  the  equipment  appropriately.  For 
example  the  system  might  evaluate  the  operator  as  being  very  expert  in  all  aspects  of  radar 
operation  and  therefore  allow  him  to  configure  the  radar  system  without  much  help  or  guidance. 
In  contrast,  the  system  may  evaluate  that  the  same  operator’s  knowledge  is  weak  in  the  area  of 
acoustic  sound  channels.  In  that  case,  the  system  might  recommend  or  select  hydrophone  depths 
for  the  sonobuoys,  while  leaving  the  operator  to  select  frequencies  and  modes. 

Consider,  for  example,  just  one  sensor  in  the  SENSO’s  suite  of  possible  sensors,  the 
passive  sonobuoy.  Sonobuoys  must  be  set  and  positioned  while  simultaneously  considering  a 
very  large  set  of  very  diverse  factors.  These  factors  can  be  grouped  into  six  categories  -  current 
mission,  physics  of  sound,  hydrophone  and  acoustic  processing  capabilities,  local  environment, 
type  of  targets  of  interest  (TOI),  and  their  likely  tactics.  For  example,  the  current  mission  could 
be  of  several  different  types  -  localizing  a  detected  target,  screening  a  transiting  battle  group, 
tracking  or  harassing  targets  passing  through  a  designated  area,  etc.  Other  mission  issues  involve 
the  likely  types  of  TOIs  and  their  probable  posture,  the  rules  of  engagement,  and  political 
constraints. 

There  are  several  principles  in  the  physics  of  sound.  The  SENSO  must  have  a  thorough 
understanding  of  different  sound  propagation  paths  and  how  they  are  affected  by  the  varying 
parameters  of  the  local  environment.  He  must  understand  how  acoustic  signals  will  be  affected 
by  their  source  emission  and  propagation  to  his  equipment.  This  includes  factors  such  as  . 
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attenuation,  Doppler  shifting,  interference,  reverberation,  multi-path  propagation,  sound 
channels,  effects  of  signal-to-noise  ratios,  etc.. 

The  SENSO  must  have  a  strong  understanding  of  the  capabilities,  limitations,  and 
opportunities  of  his  sensors  and  processing  equipment.  The  sensitivity  to  various  frequencies; 
processing  times  parameters,  and  options;  and  the  required  signal-to-noise  ratio  for  detection 
must  all  be  considered. 

The  SENSO  must  have  a  thorough  knowledge  of  the  local  environment  and  its  effects  on 
his  sensors.  In  littoral  environments,  the  environment  will  not  be  horizontally  homogenous  as  is 
often  assumed  for  deep  water.  Temperature,  salinity,  and  water  velocity  (currents)  will  all  vary 
in  three  dimensions,  as  will  bottom  topography.  Water  currents  and  cold  and  warm  water  eddies 
will  all  be  important,  if  present.  All  these  factors  will  have  a  strong  effect  on  sound  propagation. 
The  geography  (such  as  coastline,  islands,  straits,  other  choke  points,  etc.),  shipping  lanes, 
fishing  grounds  are  also  important.  Sources  of  noise  which  can  be  expected  such  as  shipping, 
drilling,  fishing,  sea  state,  storms,  wind,  seismic  activity,  and  biological  sources  should  be 
considered  during  sonobuoy  deployment. 

The  types  of  TOIs  are  important.  The  SENSO  must  have  a  thorough  understanding  of  the 
likely  sources  of  acoustic  signatures,  including  propellers,  engine,  gears,  auxiliary  pumps  and 
other  equipment,  cavitation,  resonance,  etc.  He  must  understand  when  these  signatures  are  likely 
to  be  present,  how  they  relate  to  each  other,  and  why. 

Finally  the  SENSO  must  understand  the  likely  tactics  of  his  targets.  He  must  understand 
where  and  how  deep  they  will  operate,  what  their  mission  and  objectives  are,  limitations  or 
constraints  they  must  operate  within  (E.G.  the  need  for  an  attack  submarine  to  visually  acquire 
his  target,  to  supplement  his  sonar  data),  preferred  methods  of  operation,  and  schedules  or  time 
constraints. 

This  complex  set  of  principles,  which  is  required  knowledge  for  the  SENSO,  is 
duplicated  for  each  type  of  sensor,  for  which  the  SENSO  is  responsible  -  radar,  EMS,  active  and 
passive  sonar,  SAR,  ISAR,  and  Magnetic  Anomaly  Detection  (MAD).  This  complexity  leads  to 
a  complex  definition  of  operator  proficiency.  Simple  linear  labels  such  as  Novice  through  Expert 
trivialize  a  very  complex  domain.  Instead  of  a  one-dimensional  description,  a  very  large  number 
of  dimensions  is  required  -  one  for  each  principle  or  related  set  of  principles,  which  implies 
hundreds  of  dimensions.  A  better  description  is  to  keep  track  of  the  set  of  principles  that  the 
SENSO  has  current  mastery  of  and  the  set  of  ones  on  which  he  is  weak. 

Simple  tests  of  what  principles  he  knows  and  doesn’t  know  are  not  sufficient.  Simply 
questioning  the  operator  on  the  principles  (with  multiple  choice  answers  for  example)  is  not 
sufficient,  since  what  is  most  important  is  how  the  principles  should  be  applied  in  a  tactical 
scenario.  The  operator  must  develop  a  competence  not  only  in  the  relevant  facts  and  skills,  but 
also  an  understanding  of  the  concepts  underlying  these  procedures.  Learned  principles  need  to 
be  applied  differently  in  different  situations.  For  example,  a  typical  passive  acoustic  depth 
pattern  is  deep  -  shallow  -  deep  -  shallow,  which  retains  an  ability  to  detect  submarines  operating 
both  above  and  below  the  layer.  An  operator  may  specifically  understand  and  apply  this 
principle  without  having  the  more  comprehensive  understanding.  So,  for  example,  if  a 
submarine  can  be  assumed  to  have  gone  deep  (perhaps  because  he  has  just  out-run  an  active 
sonar  tracking  situation),  an  all  deep  depth  setting  for  the  sonobuoys  might  be  more  appropriate. 
An  operator  who  has  effectively  memorized  the  principles  as  they  are  given  in  a  training  course 
may  not  have  developed  a  mental  model  allowing  him  to  make  appropriate  decisions  in 
unanticipated  situations.  He  might  continue  to  use  the  deep  -  shallow  -  deep  -  shallow  pattern  in 
the  latter  situation,  described  above.  This  example  shows  that  just  knowing  the  principle  is  not 
sufficient,  it  must  also  be  applied  correctly  and  the  best  way  to  assess  that  application  knowledge 
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is  through  scenarios  which  differentiate  between  deep  and  superficial  understanding.  If  the 
operator  assessment  focuses  on  understanding  the  operator's  cognitive  representations  of  the 
domain  and  concepts  rather  than  procedures  only,  the  operator  interface  can  be  adapted  to  him 
more  effectively.  Of  course  the  example  above  is  greatly  simplified,  but  it  presents  the  problems 
facing  operator  assessment  and  OMI  enhancement  in  decision-oriented  domains.  What  is  needed 
is  a  way  to  automatically  assess  the  knowledge  of  operators  in  this  complex  domain,  and 
automatically  reconfigure  the  sensor  equipment  for  different  levels  of  expertise  in  different  areas 

Consider  the  simplified  example  of  a  SENSO  who  does  not  understand  the  surface  duct 
phenomenon,  and  simply  assumes  the  detection  range  of  radar  surface-to-surface  is  a  constant 
range,  regardless  of  environmental  conditions.  An  automatic  assessment  system  which 
concludes  that  the  SENSO  does  not  understand  the  surface  duct  phenomenon,  may  adapt  the 
OMI  to  make  automatic  use  of  the  EMS  when  a  surface  duct  is  present  and  when  the  suspected 
location  of  the  enemy  is  outside  the  normal  detection  range  but  within  the  surface  duct  range. 

The  system  may  have  determined  this  SENSO’s  weakness  by  his  performance  on  several 
scenarios,  where  his  use  of  sensors  violated  the  recommendations  of  experts  for  the  same 
situation  and  those  scenarios  involved  the  use  of  the  surface  ducting  phenomenon. 


2.2  Artificial  Intelligence  Methodologies 

Case-Based  Reasoning 

Many  studies  have  been  performed  on  utilizing  prior  experience,  or  analogical  reasoning, 
in  various  domains  and  on  representing  prior  situational  knowledge.  Humans  reason  about  a 
given  situation  based  on  knowledge  about  that  situation  and  associations  to  previous  experiences. 
This  same  reasoning  process  applies  to  assessment  -  how  well  a  sensor  operator  will  perform  in  a 
given  situation  is  likely  to  be  similar  to  his  performance  in  similar  situations. 

Case-Based  Reasoning  (CBR)  is  the  field  of  AI  which  deals  with  the  method  of  solving  a 
current  problem  by  retrieving  the  solution  to  a  previous  similar  problem  and  altering  that  solution 
to  meet  the  current  needs.  CBR  is  a  knowledge  representation  and  control  methodology  based 
upon  previous  experiences  and  patterns  of  previous  experiences.  These  previous  experiences,  or 
"cases"  of  domain-specific  knowledge  and  action,  are  used  in  comparison  with  new  situations  or 
problems.  These  past  methods  of  solution  provide  expertise  for  use  in  new  situations  or 
problems.  From  our  previous  ITS  experience,  we  believe  that  the  general  problem  of  assessing 
operators  is  well  suited  for  the  application  of  such  a  case-based  reasoning  method. 

CBR  systems  offer  enormous  benefits  compared  to  standard  AI  approaches.  The 
knowledge  elicitation  bottleneck  is  largely  circumvented.  Cases  can  be  automatically  acquired 
directly  from  domain  experts.  Rules,  on  the  other  hand,  almost  always  require  the  intervention  of 
a  knowledge  engineer.  Instead  of  having  to  elicit  all  of  the  knowledge  required  to  derive  a 
solution  from  scratch,  only  the  knowledge  required  to  represent  a  solution  is  needed.  So 
knowledge  elicitation  is  largely  avoided  with  CBR  and  may  be  COMPLETELY  automated 
depending  on  the  type  of  application  and  the  expert.  This  makes  CBR  especially  appealing  for 
an  operator  assessment  and  OMI  enhancement  framework  that  will  potentially  be  applied  to 
multiple  domains,  because  it  reduces  the  knowledge  engineering  time  requirement. 

Conventional  knowledge  base  technology  dictates  a  single,  fixed  problem  solving 
methodology.  With  CBR,  each  case,  in  the  extreme,  can  represent  a  different  methodology.  This 
is  important  for  complex  domains  where  different  problems  or  situations,  although  sharing  the 
same  fundamental  concepts,  may  require  different  solution  strategies. 
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Stottler  Henke  Associates,  Inc.  (SHAI)  has  performed  several  projects  which  emphasized 
operator  assessment  based  on  CBR.  Many  included  simulations,  both  existing  and  new.  One 
project  dealt  with  assessing  sonar  technicians.  Our  extensive  experience  with  assessment  of 
mental  models  for  use  by  ITSs  and  our  experience  with  automated  problem-solving  can  be 
applied  to  this  project. 

We  have  developed  a  methodology  that  aids  human  experts  in  structuring  their 
experiences  for  application  to  new  problems.  This  methodology  has  proven  successful  in  a 
variety  of  domains  for  guiding  analysts  in  the  systematic  application  of  case-based  judgments  to 
new  situations.  Our  approach  to  the  development  of  a  case-based  intelligent  assessment  and 
OMI  tailoring  system  will  be  grounded  in  this  accomplishment.  We  will  extend  this  method  for 
identifying  and  formalizing  case  experience  into  a  system  for  documenting,  codifying,  and 
referencing  completed  case  analyses.  We  will  use  our  experience  in  decision-aid  design  in  the 
construction  of  a  general  intelligent  assessment  and  OMI  enhancement  system  built  on  this  case- 
based  knowledge  foundation. 

Knowledge  Elicitation 

In  developing  a  Case-Based  Reasoning  (CBR)  system  for  intelligent  assessment  and  OMI 
enhancement,  we  first  query  the  domain  experts  for  cases  (examples).  An  active  querying 
strategy  helps  us  learn  more  about  tacit  knowledge,  such  as  rapid  situation  assessment  abilities. 
Without  direct  probing,  domain  experts  may  be  unable  to  provide  explicit  motivations  for  their 
judgments.  Often  they  gain  insights  that  are  new,  even  to  themselves,  about  how  they  formed 
those  judgments  after  responding  to  the  questions.  For  these  reasons,  we  have  a  great  deal  of 
confidence  in  our  ability  to  elicit  the  knowledge  and  perceptual  and  reasoning  strategies  that  are 
necessary  components  of  a  model  that  will  be  effective  for  making  high  level  decisions. 

Our  primary  interviewing  technique  is  case-based,  dealing  with  actual  incidents  that  the 
subject  matter  experts  recall  from  experience.  By  using  this  approach,  the  experts'  actual  mental 
representations  of  their  domain  are  elicited.  All  of  these  data  are  analyzed  and  abstracted  to 
formulate  a  hierarchical  structure  representing  the  relationships  between  the  various  domain 
components. 

Each  case  elicited  will  consist  of  three  main  parts:  the  problem,  the  solution,  and  the 
process  of  deriving  the  solution,  along  with  explanations  of  each  step  of  that  process.  The 
problem  part  is  an  explanation  of  the  problem  to  be  solved  and  will  be  partly  graphical  in  nature 
to  describe  the  tactical  situation.  The  solution  will  consist  of  the  proper  SENSO  actions  to  take 
and  may  take  the  form  of  a  simple  set  of  sensor  settings,  or  be  more  complex  such  as  an 
involved  sequence  of  correct  actions  to  take  in  the  tactical  scenario.  The  solution  process  is  the 
most  complicated  part  of  all.  It  consists  of  the  steps  required  to  solve  the  problem.  With  each 
step  is  a  reference  to  the  general  principles  or  methods  used  in  that  step.  Each  reference  points  to 
a  principle  or  method  in  the  body  of  knowledge  that  the  operator  should  know.  Any  principle 
could  be  referenced  many  times  in  different  cases  but  that  principle  would  only  be  represented 
once  in  the  body  of  knowledge.  A  detailed  explanation  of  the  referenced  principles  and  problem 
solving  methods  could  be  requested  from  instructors,  thus  automatically  extending  the 
OA/OMIES.  The  cases  force  the  expert  operators  to  include  only  and  all  information  required 
for  problem  solving.  A  reasonable  organization  is  mapped  onto  unorganized  experts. 

Knowledge  Representation 

In  order  to  automate  operator  assessment  and  the  resulting  OMI  enhancement,  we  first 
established  a  representation  in  the  computer  of  the  knowledge  required.  Stottler  Henke  Associates, 
Inc.  (SHAI)  has  extensive  experience  in  selecting  AI  knowledge  representations  appropriate  for  a 
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particular  domain.  In  fact,  each  of  our  implementations  begins  with  selection  of  the  knowledge 
representation  and  definition  of  the  knowledge  (see  Related  Work).  An  appropriate  knowledge 
representation  is  one  that  naturally  and  completely  captures  desired  knowledge  in  the  domain  and 
that  can  be  successfully  and  easily  manipulated  to  meet  the  needs  of  the  application.  Using  these 
selection  criteria  and  the  knowledge  we  have  gained  from  previous  projects,  we  decided  to  use 
objects  to  represent  the  cases  of  tactical  scenarios  for  sensor  employment  which  will  be  used  for 
testing  and  enhancement  a  well.  We  have  also  found  an  object  hierarchy  useful  for  representing 
the  complex  set  of  principles  required  for  optimal  operation  of  the  sensor  suite.  The  operator’s 
mental  model  generally  is  represented  as  an  object  which  references  principles  objects  and 
performance  objects.  These  performance  objects  are  created  each  time  the  student  is  tested  on  a 
scenario.  They  record  his  actions  and  explanations  and  reference  the  case  (scenario)  on  which  he 
was  tested.  Additional  components  that  must  be  represented  are  the  sensors  themselves,  along 
with  configuration  options,  the  signal  processing  equipment  along  with  its  configuration  options, 
the  OMI  and  its  components,  the  SENSOs  tasks  and  missions,  the  environment,  the  current  tactical 
situation,  and  any  automated  tools  that  the  OA/OMIES  can  make  use  of  to  enhance  the  OMI.. 

Object  Oriented  Programming  (OOP)  is  a  methodology  for  both  representation  and 
programming.  Using  OOP  techniques,  one  can  define  different  types  of  objects  and  specialized 
program  methods  that  manipulate  them.  An  object  consists  of  slots  which  specify  the  object's 
characteristics  or  subcomponents.  Slot  values  may  be  of  several  different  types:  pointers  to  other 
tasks,  numerical  values,  Boolean  values,  lists,  or  text  strings.  Objects  can  be  connected  together 
into  a  semantic  network,  where  the  nodes  of  the  network  are  the  objects  and  the  arcs  of  the 
network  are  the  relationships  between  the  objects.  OOP  facilitates  automated  enhancement, 
where  various  object  representations  of  OMI  components  configure  themselves.  Each  object 
used  in  the  enhancement  process  has  an  associated  enhance  method  and,  in  effect,  enhances 
itself,  triggering  the  enhancement  of  its  subcomponents  or  related  components.  The  concept  of 
intelligent  entities  allows  complex  enhancement  algorithms  to  be  built  from  very  simple, 
particular  ones.  The  developer,  or  even  user,  also  has  the  capability  to  mix  and  match 
enhancement  methods  at  the  different  levels  of  the  enhancement  hierarchy  and  at  different 
components  at  the  same  level. 

Object  representations  do  not  preclude  the  use  of  other  representations,  and  in  fact, 
integrate  well  with  them.  Other  representations  which  are  useful,  in  addition  to  cases  and 
objects,  are  rules,  expert  system  technology,  Model  Based  Reasoning,  and  Fuzzy  Logic. 

Interactive  Multimedia 

Interactive  multimedia  is  used  to  both  present  the  examples  for  testing  and  to  enhance  the 
OMI,  when  required.  SHAI  has  experience  in  the  following  media: 

-  Interactive  Simulations  -  Interactive  Animations 

-  Interactive  3  Dimensional  graphics  -  Video 

-  Interactive  Photos  -  Hypertext 

-  Audio  -  Virtual  Reality 

Interactive  simulations  are  especially  useful  as  exercises  for  assessing  operator  use  of 
sensor  systems  in  tactical  situations.  For  example,  in  the  Aegis  ship  survivability  domain,  we 
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used  an  intelligent  tactical  simulation  to  test  TAO  knowledge  or  relevant  principles.  The 
operator  could  control  own  ship  sensor  configuration  and  sensor  and  weapon  system  In  our 
previous  simulations  work,  we  have  developed  tools  and  techniques  for  the  rapid  development  of 
obj  ect-oriented  simulations . 

Another  technique,  related  to  simulations  is  interactive  animations.  In  the  example 
above,  instead  of  just  printing  the  results  of  the  simulation,  the  current  sensor  contacts  and  tracks 
are  animated.  Seeing  the  incoming  missile  either  destroy  the  ship  or  be  destroyed  by  the 
intercepting  missile  makes  the  example  much  more  vivid  and  therefore  more  likely  to  be 
remembered.  SHAI  has  already  developed  a  general  three-dimensional  animation  capability. 

Interactive  3  dimensional  graphics  is  another  important  tool.  In  many  applications,  both 
tactical  and  equipment-oriented,  three  dimensional  visualization  is  required.  The  view  can  be 
rotated  by  the  operators  to  gain  a  clearer  understanding.  Other  computer  generated  graphics  such 
as  bar  charts,  pie  charts,  bitmap  files,  line  graphs,  and  plots  can  be  supported.  SHAI  has  already 
developed  an  interactive  three-dimensional  graphics  capability. 

An  extension  of  the  interactive  simulations,  animations  and  3  dimensional  graphics  is 
virtual  reality  technology.  Through  the  use  of  head  mounted  displays  or  goggles;  hand,  finger, 
and  body  tracking;  and  three-dimensional  sound,  an  operator  can  achieve  a  more  realistic, 
immersive  experience  in  the  tactical  simulation.  SHAI  is  currently  involved  in  two  virtual  reality 
projects. 

Video  can  be  especially  helpful  in  the  descriptive  section  of  a  scenario.  Videos  can  be 
made  interactive  by  allowing  user  controlled  slow  motion,  freeze  frame,  rewind,  fast-forward, 
and  branching.  Branching  is  allowed  at  certain  points  in  the  video  by  giving  the  user  a  choice 
about  which  video  among  a  set  of  choices  to  see.  SHAI  has  previously  integrated  video  clips 
into  our  software. 

Interactive  photos  also  lend  vividness  to  presentations  and  examples.  A  photo  is 
interactive  if  can  be  zoomed  or  panned.  Certain  regions  or  annotations  may  be  mouse-sensitive 
to  allow  further  information  to  be  presented  such  as  hypertext  or  other  photos.  SHAI  has 
implemented  a  prototype  zoom  capability  in  our  targeting  project  (see  Related  Work). 

Hypertext  is  mouseable  text  with  further  information  available  on  mouseable  words  or 
topics.  This  further  information  may  also  be  hypertext  or  some  other  kind  of  media.  SHAI  has 
developed  several  systems  utilizing  hypertext. 

Another  possible  media  is  audio.  Audio  can  be  sound  recording  to  add  realism  to  an 
example  or  might  be  recorded  or  generated  voice  which  describes  the  tactical  situation  or  asks 
questions  as  to  the  rationale  for  operator  actions  in  it.  Audio  can  be  made  interactive  in  the  same 
ways  as  video.  SHAI  has  previously  utilized  computer  generated  speech  for  a  project  to  help 
nonvocal  quadriplegics  communicate. 
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3.0  Phase  I  Objectives/Tasks 

Section  3.1  gives  the  original  Phase  I  technical  objectives  listed  in  the  original  Phase  I 
proposal  (all  of  which  were  accomplished)  and  Section  3.2  describes  the  tasks  of  this  phase  I 
effort. 


3.1  Phase  I  Objectives 

All  Phase  I  objectives  described  in  the  Phase  I  Proposal  were  accomplished  and  are 
reproduced  below.  In  Phase  I,  we  investigated  a  case-based  reasoning  approach  to  intelligent 
Operator  Assessment  and  Operator  Machine  Interface  Enhancement  Systems  (OA/OMIESs). 

We  developed  a  prototype  case-based  OA/OMIES  within  the  LAMPS  SH-60R  MMHU  ASW 
domain.  We  determined  the  requirements,  both  hardware  and  software,  for  integrating  the 
OA/OMIES  with  existing  systems.  Specifically,  there  were  five  Phase  I  objectives  listed  in  the 
proposal  and  approved  at  the  kick-off  meeting  which  are  listed  below.  The  Phase  tasks  and 
results  are  further  described  in  more  detail  below  in  Sections  3.2  and  4.0. 

1.  Identified  LAMPS  SH-60R  MMHU  Assessment  and  OMI  Enhancement 
Requirements:  Working  closely  with  the  Navy,  we  identified  a  specific  subset  of  the  MMHU 
domain  for  which  operator  assessment  and  OMI  enhancement  must  consider  the  cognitive 
abilities  of  the  operator. 

2.  Developed  Strategies  for  Mental  Model  Assessment:  Through  intelligent  indexing  of 
scenarios  and  other  techniques,  we  developed  general  analytical  routines  for  assessing  an 
operator's  mental  model. 

3.  Developed  Strategies  for  OMI  Enhancement.  Given  an  accurate  mental  model  of  the 
operator,  we  determined  how  the  OMI  can  and  should  be  altered.  One  method  was  to  use  CBR 
applied  to  expert  scenarios  of  sensor  employment.  Another  was  to  develop  rules,  methods,  or 
algorithms  based  on  the  set  of  principles  in  which  the  operator  is  weak.  Various  metrics  were 
investigated  and  developed. 

4.  Case-Based  Representation  and  Reasoning  Architecture:  We  produced  a  generic 
architecture  for  the  case-based  OA/OMIES.  The  benefits  of  CBR  for  both  automated  assessment 
and  intelligent,  operator-tailored  adaptation  of  the  OMI  were  demonstrated  through  this  system. 

5.  Prototype  Development:  We  developed  a  proof-of-concept  prototype  on  a  PC,  based 
on  the  system  architecture.  The  prototype  demonstrated  important  CBR  functionality  both  as  a 
general  assessment  and  OMI  enhancement  system  and  as  an  OA/OMIES  implementation  within 
a  subset  of  the  LAMPS  SH-60R  MMHU  domain.  This  aided  us  in  the  prediction  of 
computational  requirements  of  the  final  system. 


3.2  Phase  I  Tasks 

An  eight  task  approach  was  proposed  for  accomplishing  the  Phase  I  research  objectives. 
The  tasks  were  to: 

1.  Select  the  subset  of  the  LAMPS  SH-60R  domain  to  focus  the  study 

2.  Define  the  preliminary  case  structure  for  the  elicitation  procedure 
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3.  Conduct  knowledge  elicitation 

4.  Design  the  case  base  structure  and  retrieval  methods 

5.  Investigate  Techniques  for  OMI  enhancement 

6.  Investigate  the  integration  requirements 

7.  Implement  OA/OMIES  prototype 

8.  Prepare  the  Phase  II  OA/OMIES  design  and  final  report 


Task  Descriptions 

1 .  Select  the  subset  of  the  LAMPS  SH-60R  domain  to  focus  the  study:  Working  in 
conjunction  with  Navy  representatives,  we  selected  a  representative  subset  of  the  LAMPS  SH- 
60R  domain  for  our  feasibility  study.  We  chose  to  concentrate  on  the  sensor  operator  and,  more 
specifically  for  the  prototype,  on  ASW  situations.  We  did  include  tasks  normally  associated  with 
the  ATO  as  well.  The  results  are  given  in  Section  4.0. 

2.  Define  Preliminary  Case  Structure:  We  determined  an  appropriate  representation  for 
cases  in  the  domain.  The  cases  consisted  of  attributes  for  describing  the  problem-solving 
principles  and  methods  as  well  as  their  explanations.  We  also  examined  potential  similarity 
metrics  and  retrieval  methods. 

3.  Conduct  knowledge  elicitation  to  develop  the  domain  model:  Based  on  the  research  into 
the  general  qualities  of  mental  models,  SHAI  elicited  knowledge  from  experts  in  the  SH-60 
ASW  mission.  We  applied  a  cognitive  task  analysis  approach,  where  the  critical  decisions  were 
identified  and  the  factors  and  issues  which  must  be  considered  were  elicited.  We  made  strong 
use  the  knowledge  acquisition  method  -  Method  of  Cases,  where  experts  ran  through  examples 
from  their  experience. 

Our  thesis  was  that  assessment  and  OMI  enhancement  strategies  would  be  primarily  case- 
based,  because  the  complexity  of  the  required  operator  mental.  Domain  experts  were 
interviewed  individually  and  presented  with  problems  to  determine  their  mental  model  by 
recording  their  situational  performance  both  in  previous  experiences  and  new  scenarios.  The 
experts'  knowledge  was  used  to  develop  a  quality  representation  of  the  trial  domain,  by  which 
operators'  mental  models  will  be  measured.  Typically  Case-Based  Reasoning  (CBR)  techniques 
can  be  used  effectively  for  knowledge  acquisition  in  case-based  applications,  and  this  was 
effective  for  this  domain  as  well.  Thus,  a  CBR  knowledge  elicitation  component  will  likely  be 
employed  in  the  Phase  II  OA/OMIES  based  on  the  findings  in  the  Phase  I  elicitation. 

4.  Design  the  structure  for  the  case  base  and  retrieval  methods:  We  produced  a  generic 
architecture  for  the  case-based  OA/OMIES  which  is  described  in  Section  5.0.  The  benefits  of  a 
case-based  approach  for  automated  knowledge  acquisition,  intelligent  assessment,  and  operator- 
responsive  OMI  enhancement  were  demonstrated  through  this  system.  The  case  structure  was 
capable  of  representing  an  example  (scenario)  which  included  the  problem  (tactical  situation),  its 
solution  (sensor  system  configurations  and  actions)  and  an  explanation  of  the  solution  which 
referenced  general  principles  and  methods.  An  object-oriented  approach  was  used  to  represent  a 
case.  We  used  object  structures  to  provide  a  framework  for  knowledge  representation  and 
program  control.  As  the  case  structure  was  defined,  a  retrieval  method  was  outlined  for  the 
system.  It  is  the  intelligent  retrieval  which  served  as  the  primary  driver  of  the  operator 
assessment,  followed  by  an  analysis  of  the  operators  performance. 
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5.  Investigate  Techniques  for  OMI  enhancement 

SHAI  investigated  techniques  to  use  the  operator  model  to  enhance  the  user  interface,  in  a 
way  which  is  customized  to  the  particular  operator  and  which  optimizes  the  combined 
operator/sensor  system  performance.  This  enhancement  investigation  included  both  the  types  of 
alterations  to  be  made  as  well  as  how  those  alterations  can  be  accomplished  automatically.  The 
types  of  alteration  included  automatically  setting  sensor  operation  or  processing  modes, 
parameters,  options,  etc.;  priming  certain  help  files  or  features  for  the  operator;  recommending 
certain  configuration  settings;  starting  and  initializing  decision  aids  for  the  operator;  or  making 
use  of  expert  systems  to  configure  the  equipment  appropriately.  We  investigated  a  number  of 
enhancements,  including  use  of  rule-based  or  other  knowledge  based  systems;  use  of  the  cases 
acquired  from  experts;  and  other  technologies. 

6.  Investigate  the  integration  requirements:  We  worked  with  the  Navy  to  determine  the 
hardware  and  software  requirements  for  integrating  the  OA/OMIES  with  the  new  hardware  and 
software  systems  being  developed  for  the  SH-60R  and  for  placing  the  OA/OMIES  onboard  to 
support  OMI  enhancement. 

7.  Implement  OA/OMIES  prototype:  We  developed  a  prototype  OA/OMIES  for  the 
LAMPS  SH-60R  MMHU,  to  provide  an  architecture  for  evaluating  the  feasibility  of  an  case- 
based  OA/OMIES.  This  prototype  incorporated  the  strategies  developed  earlier  for  intelligent 
operator  mental  model  assessment  and  OMI  enhancement.  It  was  designed  for  easy  application 
to  new  domains.  This  prototype  provided  a  sample  of  the  "look  and  feel"  of  the  system  and 
contained  representative  CBR  functionality  that  operated  on  the  chosen  subset  of  the  LAMPS 
SH-60R  domain  (ASW  Sensor  Operations).  It  was  used  to  demonstrate  a  specific  application  of 
the  use  of  examples  in  assessment  and  OMI  enhancement.  It  also  demonstrated  the  ability  of  the 
system  to  automatically  retrieve  similar  examples,  and  to  modify  sensor  equipment  OMIs  to 
meet  current  operator  needs.  While  initial  evaluation  of  the  prototype  was  be  carried  out  in 
Phase  I,  its  primary  use  will  be  in  Phase  II  for  more  comprehensive  testing  of  the  assessment  and 
OMI  enhancement  strategies  and  possible  exploration  of  other  domains.  It  is  described  in  more 
detail  in  Section  6. 

8.  Prepare  the  Phase  II  OA/OMIES  design  and  final  report:  This  final  report  describes  the 
development  and  architecture  of  both  the  general  and  the  specific  case  structures  and  retrieval 
methods  and  includes  the  Phase  II  design,  in  Section  5.0.  This  design  includes  the  architecture 
for  all  modules.  The  evaluation  of  the  prototype  in  its  trial  domain  is  presented.  A  future 
research  section  outlines  the  requirements  needed  to  develop  the  full-scale,  general  intelligent 
OA/OMIES,  for  use  on-board  the  SH-60R. 
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4.0  Phase  I  Accomplishments/Results 

This  project  really  got  underway  at  the  kick-off  meeting  which  was  held  March  18th.  It 
was  decided  there  to  tend  to  concentrate  on  the  sensor  operator,  since  that  position  appeared  to  be 
fluctuating  less.  We  did  review  several  documents  (SH-60F  NWP,  OEC  for  SH-60R,  LAMPS 
MK  III  Block  II  Upgrade  Design  Description  Document,  and  the  HEDAD-0 ).  A  choice  within 
the  two  broad  domains  of  the  MMR  and  acoustic  systems  appeared  to  be  the  most  promising, 
since  these  appear  to  require  the  most  operator  knowledge,  be  the  most  difficult  set  of  tasks,  and 
relate  to  the  aircraft’s  two  primary  missions.  Two  possible  choices,  because  SHAI  already  had 
some  experience  in  these  areas,  were  to  concentrate  on  passive  acoustic  detection  and 
classification  or  passive  sonobuoy  placement  decisions.  We  did  end  up  choosing  acoustic 
systems  on  which  to  concentrate.  More  specifically,  the  subset  chosen  during  the  recent  visit  was 
the  union  of  domains  of  two  operator-machine  interface  problems,  each  of  which  was  taken  from 
a  'DSS  wish  list'  composed  by  VX-1.  One  problem  was  the  multitude  of  processor  mode 
combinations  during  passive  acoustic  search,  for  which  our  system  should  provide  sensor 
settings  recommendations  to  an  inexpert  or  overwhelmed  operator.  The  other  problem  was  the 
inability  to  associate  contacts  on  multiple  sensor  types,  which  could  be  solved  by  having  our 
system  provide  alerts  to  contact  information  not  being  viewed  by  the  operator,  when  appropriate. 
These  two  problems  guided  our  development  of  a  knowledge  base  and  interface  enhancement 
prototype. 

On  28  May  we  met  with  Russ  Hallauer  to  discuss  the  SH-60R  domain,  specifically  in  the 
context  of  the  year  2005  scenarios.  This  discussion  resulted  in  broad  tactical  and  operational 
knowledge,  as  well  as  contacts  for  further  investigation  of  the  platform's  usage  in  detail.  The 
following  day  we  visited  VX-1  at  Patuxent  River,  where  we  established  expert  contacts  and 
tentative  plans  to  observe  their  development  of  the  next  scenario,  and  discuss  with  them  low- 
level  considerations  of  the  original  six  scenarios. 

During  the  week  of  1 1  August  we  did  visit  VX-1  at  Patuxent  River  and  met  with  several 
expert  contacts.  A  working  group  meeting  served  to  better  define  the  scope  and  target 
functionality  of  the  Phase  I  effort,  and  individual  meetings  with  experts  have  given  us  a  good 
deal  of  knowledge  of  the  relevant  domain.  Our  discussions  also  revealed  the  applicability  of 
various  interface  adaptation  techniques,  as  suggested  and  evaluated  by  expert  aircrew.  We 
obtained  several  documents  detailing  the  tactical  and  technical  operation  of  the  SH-60R  and  its 
various  sensor  and  weapon  systems,  and  we  visited  the  Replacement  Air  Groups  at  North  Island 
in  order  to  observe  training  on  the  SH-60B  and  SH-60F  platforms.  The  knowledge  gathered 
during  this  visit  was  sufficient  for  us  to  complete  a  more  detailed  design  and  begin  definition  and 
implementation  of  the  system’s  computational  architecture.  A  preliminary  design  of  the  system  is 
given  in  Section  5.0.  Further  research  and  a  visit  to  North  Island,  allowed  us  to  flesh  out  our 
operational  knowledge  base,  as  well  as  resolve  remaining  low-level  implementation  and  interface 
details. 
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On-board  Fielding  Issues 

The  OA/OMIES  will  be  highly  feasible  to  field  on-board  the  SH-60R  for  several  reasons. 
Using  the  laptop  concept  saves  space,  weight  and  power.  Typical  laptops  with  12  “  screens 
weigh  about  7  pounds  (extra  hardening  may  add  some  small  amount  of  weight),  take  up  less  than 
a  tenth  of  a  cubic  foot  when  closed  (8.5”  x  11”  x  1.75”),  have  small  operational  dimensions  (8.5” 
x  11”  x  10”,  open),  and  use  little  power  (20  Watts).  The  computational  power  and  space 
requirements  of  our  approach  our  modest,  allowing  the  use  of  existing  Pentium  laptops,  which 
easily  provide  the  needed  computational  power.  The  power  interface  would  tend  to  be  a  simple 
low-power  DC  connection,  as  required  by  the  laptop  manufacturer.  Worse-case  a  small, 
inexpensive  transformer  would  be  required  to  change  the  on-board  voltage  for  the  laptop.  The 
data  interface  would  be  via  a  standard  RS-232  connection.  The  bandwidth  requirements  of  the 
connection  are  very  low,  with  the  possible  exception  of  sensor  images.  These  would  only  be 
needed  if  the  enhancement  system  included  classification  aids,  running  on-board  the  laptop. 

Furthermore,  the  OA/OMIES  can  be  implemented  in  a  highly  efficient  and  modularized 
manner.  We  estimate  less  than  40,000  software  lines  of  code  (SLOC),  based  on  previous 
implementations  that  we  have  performed  at  similar  levels  of  effort.  This  SLOC  will  be  divided 
up  into  several  very  separate  modules  for  easier  implementation  and  testing.  First,  the 
assessment  and  enhancement  modules  are  so  separate  that  they  will  be  running  on  different 
processors  and  at  different  times.  Second,  the  domain  knowledge  is  separate  from  the  scenario 
and  case  knowledge  which  is  separate  from  the  software  which  makes  use  of  it.  As  described  in 
Section  5.2  the  assessment  and  enhancement  modules  are  also  divided  into  smaller,  very  separate 
components.  Finally,  the  most  critical  software,  that  actually  runs  on-board,  does  not  include  the 
assessment  system  and  is  mostly  made  up  of  very  separate  enhancement  codes,  as  described 
below  and  in  Section  5.2. 
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5.0  Phase  II  OA/OMIES 
5.1  System  Functionality 

General 


The  Operator  Assessment/OMI  Enhancement  System  (OA/OMIES)  will  assess  an 
operator’s  knowledge  and  tactical  proficiency  by  testing  him  with  example  tactical  scenarios,  off¬ 
line.  An  example  consists  of  a  problem  description,  solution,  and  explanation  or  steps  leading  to 
the  solution.  An  exercise  is  extracted  from  an  example  by  only  showing  the  operator  the 
problem.  He  must  then  generate  the  solution  himself.  His  solution  and  solution  steps  can  be 
compared  to  that  of  the  exercise  for  grading,  deficiency  diagnosis,  and  interface  alteration. 

The  system  works  interactively  with  the  operator  to  test  his  knowledge  by  using  scenarios 
of  sensor  employment  in  tactical  situations.  These  scenarios  are  generally  presented  through  a 
tactical  simulation.  But,  they  may  also  be  presented  by  simply  explaining  the  situation  to  the 
student  and  getting  back  from  him  how  he  would  operate  his  sensor  equipment  in  that  situation. 
This  approach  was  taken  in  a  project  to  assess  Sonar  Technicians  (STs)  in  their  ability  to 
configure  their  processing  equipment  and  analyze  LOFARGRAMS.  The  tactical  situation  was 
presented  to  the  ST,  he  decided  how  to  configure  the  equipment,  and  the  corresponding 
LOFARGRAM  was  displayed.  This  often  created  additional  opportunities  for  processing 
configuration  choices. 

In  order  to  tailor  the  equipment  operation  to  the  individual  operator,  we  will  keep  a  model 
of  each  operator  tested  using  the  OA/OMIES.  The  operator  model  will  contain  the  operators' 
actions  and  decisions  during  different  exercises,  the  principles,  procedures,  and  techniques  which 
have  been  tested,  and  those  that  have  been  mastered  based  on  performance  on  exercises.  The  set 
of  principles,  procedures,  rules,  and  tools  referenced  in  the  solutions  of  problems  the  operator  has 
solved  successfully  represent  the  operator's  mastered  skills.  Based  on  the  pattern  of  his 
unsatisfactory  performance  on  exercises,  a  set  of  topics  and  principles,  or  combinations  of  them, 
can  be  developed  which  form  a  hypothesis  as  to  what  knowledge  the  operator  does  not 
understand.  This  hypothesis  is  the  basis  for  the  operator  model  which  will  be  used  to  enhance 
the  user  interface  to  counteract  his  deficiencies.  The  operator  model  can  also  be  referenced  by  a 
supervisor  or  Intelligent  Tutoring  System  (ITS)  to  monitor  the  operator's  abilities  and 
weaknesses  and  attempt  to  remediate  them.  If  an  ITS  is  used,  cases  which  have  been  stored  for 
testing  can  be  used  to  help  remediate  him.  The  operator  model  will  be  high  fidelity  and  reflect 
the  skills,  knowledge,  and  error-rate  of  the  operator.  The  model  will  evolve  in  size  and 
complexity  as  the  skills  and  knowledge  of  the  operator  increase. 

The  OA/OMIES  will  then  make  use  of  the  operator  model  to  enhance  the  user  interface, 
in  a  way  which  is  customized  to  the  particular  operator  and  which  optimizes  the  combined 
operator/sensor  system  performance.  This  enhancement  may  include  automatically  setting 
sensor  operation  or  processing  modes,  parameters,  options,  etc.;  priming  certain  help  files  or 
features  for  the  operator;  recommend  certain  configuration  settings;  starting  and  initializing 
decision  aids  for  the  operator;  or  making  use  of  expert  systems  to  configure  the  equipment 
appropriately.  The  enhancement  can  be  performed  in  a  number  of  ways,  all  of  which  were 
investigated  and  possibly  will  be  implemented  in  parallel.  Since  the  operator  model  includes  the 
principles  and  skills  in  which  the  operator  is  weak,  these  are  passed  to  the  on-board  enhancement 
system  to  set  the  OMI  appropriately  for  the  given  circumstances.  For  example,  if  the  operator  is 
weak  in  the  concept  of  the  shallow  sound  layer  for  acoustic  signals,  then  a  rule  might  set  the 
hydrophone  depths  for  the  sonobuoys,  automatically,  based  on  the  current  layer  depth  and 
thickness,  or  display  the  appropriate  recommendations.  In  a  more  complex  situation,  the  rule 
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might  call  an  entire  expert  system,  to  calculate  and  set  sensor  parameters.  The  list  of  sensor 
settings  associated  with  each  principle  could  also  be  generated  automatically  from  the  cases  in 
the  system,  since  each  action  (such  as  setting  a  sensor  parameter)  has  an  attached  description 
which  includes  references  to  principles. 

Another  opportunity  exists  for  adapting  the  interface.  Since  the  cases  include  optimum 
sensor  settings,  according  to  an  expert,  they  can  be  used  to  set  the  sensor  equipment 
configuration.  Cases  which  are  similar  to  the  current  tactical  situation  can  be  retrieved.  In  those 
cases,  sensor  configuration  settings  that  required  use  of  principles  the  operator  is  weak  in,  could 
be  used  as  a  basis  to  set  the  sensor  configuration  in  the  current  situation.  Multiple  similar  cases 
could  be  retrieved,  if  the  case-base  is  sufficiently  dense,  to  confirm  the  correctness  of  these 
settings.  For  example,  the  situation  may  be  to  relocate  a  nuclear  submarine  that  has  escaped  an 
active  tracking  attempt.  A  similar  case  is  retrieved  from  the  case-base  of  expert  entered 
scenarios.  This  similar  past  scenario  may  involve  trying  to  reacquire  a  nuclear  submarine  after 
an  unsuccessful  attack,  in  similar  acoustic  conditions.  The  expert,  in  that  case,  has  set  his 
sonobuoys  on  the  deep  setting  with  an  explanation  that  references  the  principle  that  nuclear 
submarines  typically  run  fast  and  deep  when  detected.  Say,  for  this  example,  that  when  the 
system  examines  the  operator’s  mental  model  it  finds  that  the  student  is  either  weak  in  his 
understanding  of  nuclear  submarine  tactics,  or  this  principle  in  particular.  It  then  uses  the 
expert’s  deep  sonobuoy  setting  as  a  basis  for  determining  that  the  deep  setting  is  also  applicable 
in  this  situation  and  since  the  operator  is  deemed  weak  in  this  area,  the  deep  setting  is 
recommended  for  the  sonobuoys. 

Another  way  the  cases  can  be  used  is  to  scan  the  cases  similar  to  the  current  situation,  for 
references  to  principles  that  the  operator  is  weak  in.  Any  sensor  configuration  settings  which 
reference  those  principles  should  be  set  automatically  to  defaults  as  calculated  by  rules,  expert  or 
knowledge-based  systems,  or  comparison  to  similar  cases. 

A  case-based  OA/OMIES  can  monitor  the  operator's  actions  in  simulations  and  analyze 
them  with  respect  to  the  different  aspects  of  a  domain.  For  example,  during  a  simulation,  a 
sensor  operator  might  configure  the  acoustic  processing  system  to  have  very  long  integration 
time  (perhaps  8  minutes)  in  a  situation  when  it  is  not  appropriate  (such  as  attempting  to  track  a 
fast-moving  target),  but  perform  the  procedure  of  setting  the  integration  time  correctly 
nonetheless.  The  operator's  performance  needs  to  be  analyzed  to  determine  the  correspondence 
between  the  different  components  of  his  mental  model  and  those  of  the  domain.  While  the  steps 
involved  in  the  integration  time  setting  procedure  may  be  understood,  the  tactical  decision  of 
when  to  do  so  needs  to  be  an  identified  weakness.  An  OA/OMIES  can  easily  identify  this 
deficiency  by  testing  the  student  on  passive  acoustic  scenarios.  During  actual  OMI 
configuration,  based  on  this  weakness,  the  OMI  may  be  configured  such  that  the  integration  time 
is  set  at  a  more  appropriate  default,  such  as  1  to  2  minutes. 

Overview 


After  reviewing  the  HEDAD-0  document,  with  more  concentration  on  the  Sensor 
Operator,  it  appeared  that  direct  manipulation  of  the  SH-60  interface  may  be  inadvisable. 

Starting  from  the  ideas  presented  to  SHAI  at  the  kick-off  meeting  we  are  proposing  the  following 
concept:  The  sensor  operator  is  provided  an  additional  display,  ostensibly  for  the  display  of  help 
files.  On-board  the  Space  Shuttle,  where  similar  integration,  safety,  and  mission  critical 
concerns  exist,  they  often  use  laptop  computers  for  similar  functions  as  those  described  here. 

That  might  be  a  possibility,  though  not  a  requirement  for  our  approach,  for  the  SH-60R.  This 
additional  display  would  be  under  the  control  of  the  SHAI  OMI  Adaptation  software.  It  could 
monitor  the  operator  actions  and  the  outputs  and  states  of  various  systems  through  the  same 
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interface  as  the  DSS  or  its  own  RS-232.  The  Adaptation  software  roughly  needs  the  same  set  of 
data  that  is  displayed  to  the  operator,  with  the  possible  exception  of  sensor  images,  which  would 
only  be  needed  if  there  will  be  automatic  classification  aids.  I  believe  an  important  and  powerful 
display  concept  is  to  minimize  the  spontaneous  presentation  of  information  on  this  display  for  3 
reasons.  1)  This  additional  display  is  inherently  auxiliary  in  nature;  it  is  not  the  primary  interface 
to  the  operator.  2)  As  such,  it  will  be  a  distraction  from  the  primary  displays  and  should, 
therefore,  be  used  sparingly,  only  when  positive  proof  exists  that  there  is  a  good  reason  to.  And 
3)  having  the  display  primarily  blank,  only  showing  information  when  the  system  knows  that  the 
operator  needs  it,  will  teach  the  operators  that  when  something  is  displayed,  he  needs  to  pay 
attention  to  it,  that  it  is  addressing  a  relevant  knowledge  deficit  that  they  have.  This  will  keep  the 
operators  from  learning  to  ignore  these  spontaneous  helps  and  advice. 

The  visualized  sequence  whereby  the  OMI  Adaptation  software  will  be  utilized  consists 
of  3  primary  phases.  The  first  is  evaluation.  In  the  evaluation  phase,  the  system  tests  operator 
knowledge  and  builds  a  model  of  his  knowledge.  Much  of  this  would  occur  by  testing  him  with 
a  tactical  simulation  and  analyzing  his  responses.  The  amount  of  detailed  knowledge  required  to 
operate  the  SH-60R  may  preclude  testing  all  knowledge  through  scenarios.  Some  device 
operation  specific  details  may  need  to  be  tested  though  a  multimedia  question  and  answer  format. 
In  one  project  we  are  performing,  operator  stations  in  a  CIC  are  rendered  in  a  virtual 
environment.  Something  similar  could  be  done  to  efficiently  and  accurately  question  the 
operator  about  device  operation  knowledge.  After  evaluation,  the  identified  deficiencies  could 
be  fed  to  a  training  system  which  could  try  to  remediate  them.  Reevaluation  should  then  occur  to 
update  the  operator’s  knowledge  model.  In  the  second  stage,  the  OMI  Adaptation  system  would 
present  proactively,  minimal  tailored  information  on  the  auxiliary  display  during  an  actual 
mission.  This  mission  might  be  either  a  training  mission  or  actual  combat.  Finally,  especially  if 
a  training  mission  was  performed,  the  third  phase  could  consist  of  a  debriefing. 

SH-60R  OMI  Adaptation  and  Display  Concepts 

There  are  several  interface  adaptations  that  could  be  made  on  the  auxiliary  display.  One 
is  that  Help  information  appears  when  the  operator  is  in  either  a  situation  he  is  known  to  be  weak 
in  or  accessing  equipment  or  modes  which  he  is  deficient  in.  Usually  this  is  a  blank  screen  -  it 
ONLY  displays  information  when  the  system  knows  he  needs  it.  Therefore  it  could  be  very 
powerful  and  not  distracting.  If  text  appears,  it  is  probably  something  he  doesn’t  know.  Of 
course  he  could  always  request  help  when  the  screen  is  blank,  or  navigate  around  a  hypertext 
help  system. 

Another  form  of  adaptation  is  to  make  suggestions  as  in  an  Expert  System,  SO’s 
Associate  concept.  Again,  the  system  would  filter  the  suggestions  so  that  the  expert  system  only 
provide  information  when  the  operator  is  known  to  be  weak  in  the  area  (or  he  explicitly  asks  for 
it).  This  advice  might  include  which  displays  are  appropriate  for  which  tactical  situations,  since 
there  are  several  choices  and  it  has  a  huge  effect  on  situation  awareness. 

An  interesting  specialization  of  this  concept  is  classification  aids  (if  he’s  weak  in  one  of 
those  areas).  Possible  ideas  are  an  ISAR  images  case  base,  ISAR  classification 
principles/procedures,  advice  on  acoustic  analysis  such  as  which  processing  and  display  options 
to  use,  suggested  buoy/dipper  pattems/types/configurations,  etc. 
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Another  type  of  adaptation  involves  recreation  of  an  existing  display  (that  would 
typically  appear  on  his  primary  display  as  described  by  the  HEDAD-O).  Adaptations  include 
highlighting  something  that  the  operator  has  shown  he  frequently  misses;  indicating  that  this  is 
the  display  that  the  operator  should  be  looking  at  (when  he’s  weak  at  deciding  or  something 
important  has  shown  up);  changing  the  display  for  operator  identified  weaknesses  such  as 
allowing  for  differences  in  cognitive  capacities  (e.g.  degree  of  complexity);  and  making  it  better 
for  a  particular  operator  -  e.g.  overcoming  sensor  envelope  display  limitations  and  more  complex 
operator-tailored  decluttering. 

The  system  could,  as  mentioned  earlier,  provide  off-line  training.  This  could  take  the 
form  of  a  unique  form  of  just-in-time  training,  which  is  both  mission  and  operator  specific.  The 
training  system  could  exercise  him  in  areas  both  that  he’s  weak  in  and  that  are  required  for  the 
mission.  It  also  allows  the  system  to  test  him  on  the  important  (for  that  mission)  issues  one  last 
time. 


5.2  Design 
5.2.1  Summary 

The  Operator  Assessment  and  Operator  Machine  Interface  Enhancement  System 
(OA/OMIES)  is  composed  of  two  major  subsystems:  the  assessment  module,  which  determines 
the  areas  of  the  operator's  expertise  and  generates  a  mental  model  for  that  operator,  and  the 
enhancement  module,  which  makes  use  of  the  mental  model  to  enhance  the  operator's  interaction 
with  the  SH-60R.  A  core  knowledge  base  underlies  both  modules,  composed  of  a  hierarchy  of 
principles  that  capture  the  expertise  crucial  to  proficient  crewman  performance  in  all  areas  of  the 
SH-60R  operational  domain.  This  knowledge  base  is  associated  with  a  case  base  of  mission 
scenarios,  describing  in  each  case  the  expertise  necessary  to  proper  understanding  of  and 
performance  in  that  scenario.  The  case  base  forms  a  set  of  simulation  scenarios  to  be  used  in 
operator  assessment,  and  also  allows  for  case-based  retrieval  of  scenarios  in  support  of  real-time 
machine  interface  enhancement.  The  overall  design  is  shown  in  Figure  1 . 
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Figure  1.  OA/OMIES  Overall  System  Design 


Knowledge  Base 

The  knowledge  base  used  in  the  OA/OMIES  is  a  hierarchical  breakdown  of  interrelated 
principles  capturing  expertise  in  SH-60R  operations  in  the  ASW  and  ASST  domains.  Table  1  is  a 
partial  list  of  the  highest  levels  of  the  hierarchy. 


Table  1.  High-level  Principle  Hierarchy 


•  physics 

acoustic 

ocean  layering 
Doppler 
harmonics 
littoral  effects 
electromagnetic 
ducting 

•  equipment  (  repeated  for  all  sensors, 
weapons,  consoles) 

proper  operation 
configuration 
capabilities 
limitations 
abnormal  operations 
interpretation  of  data 
classification 


•  mission  (repeated  for  ASW  &  ASST) 

situational  awareness 
tactics 

execution  of  mission 
search 
localization 
tracking 
attack 
covertness 
team  coordination 
division  of  labor 
external  communications 
ownship  coordination 
scene  of  action  command 

•  enemy 

platforms 

capabilities 

signatures 

tactics 
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The  lower  levels  of  the  complete  hierarchy  are  composed  of  general  and  specific 
principles  which  include  the  operational  knowledge  necessary  to  determine  the  appropriate  action 
for  a  crewman  to  take  in  a  particular  set  of  circumstances.  For  instance,  in  the  initial 
implementation  of  the  search  phase  of  an  ASW  mission,  the  mission  principle  of  maintaining 
covertness  suggests  the  use  of  acoustic  buoys,  principles  of  sensor  limitations  and  presumed 
capabilities  of  the  submarine  suggest  a  particular  sonobuoy  placement  pattern  and  depth  settings 
to  optimize  detection,  principles  of  situational  awareness  and  console  operation  suggest 
monitoring  of  certain  sensors  on  certain  displays,  and  principles  of  sensor  data  interpretation 
suggest  the  outcome  of  localization  and  classification.  As  events  unfold,  various  principles  come 
into  play  suggesting  the  appropriate  conclusions  and  reactions  resulting  from  new  data  and 
circumstances. 

The  principle  hierarchy,  and  associated  knowledge  about  proper  application  of  principles 
to  scenarios,  provides  a  facility  for  detecting  at  any  time  in  a  mission  a  correct  action  (e.g., 
evasive  maneuvers)  and/or  determining  a  correct  conclusion  (e.g.,  target  submarine  is  deeper 
than  expected).  The  goal  of  the  OA/OMIES  is  to  detect  situations  in  which  a  particular  operator 
might  be  lacking  relevant  expertise,  and  in  those  situations  enhance  the  machine  interface  to  aid 
performance  of  the  most  appropriate  action  or  inform  the  operator  of  an  appropriate  assumption. 

Assessment 


The  knowledge  base  described  above  can  be  considered  the  expert  model  to  which 
operators  are  compared,  although  the  case-based  approach  used  in  the  OA/OMIES  circumvents 
the  need  for  explicit  construction  of  an  expert  model.  The  purpose  of  the  assessment  module  of 
the  OA/OMIES  is  to  generate  a  mental  model  for  each  operator,  formed  as  an  annotation  of  the 
principle  hierarchy  describing  that  crewman's  deficiencies  in  understanding.  In  general  the 
system  will  determine  whether  or  not  each  operator  has  an  understanding  of  each  principle,  from 
the  general  (underwater  sound  propagation  in  littoral  environments)  to  the  specific  (how  to 
configure  a  particular  weapon  fire  control  solution).  Because  this  determination  is  performed 
through  the  simulation  of  scenarios,  deficiencies  are  identified  in  an  operational  sense,  as 
opposed  to  explicit  knowledge  tests  which  detect  only  declarative  ("textbook")  knowledge. 
Cognitive  psychology  has  produced  extensive  evidence  that  a  person  may  have  extensive  capture 
of  the  latter,  and  yet  be  unable  to  apply  that  knowledge  effectively,  as  with  the  former.  While  the 
OA/OMIES  could  be  used  for  conventional  instruction  in  training  environments  or  in  flight 
debriefing,  it  is  essential  that  interface  enhancement  be  sensitive  to  an  operator's  ability  to 
employ  operational  knowledge  in  realistic  situations. 

Case  Base 

Assessment  of  an  operator's  expertise  is  performed  automatically  by  observation  of  the 
performance  of  the  operator  in  mission  simulations.  For  this  purpose  the  OA/OMIES  draws  upon 
a  case  base  of  mission  situations,  each  case  representing  a  particular  set  of  possible 
circumstances.  The  cases  are  associated  with  sets  of  principles  which  comprise  the  expertise 
necessary  to  perform  correctly  in  those  circumstances,  and  they  themselves  represent  the  correct 
actions  that  those  principles  indicate.  In  contrast  to  an  actual  expert  system  which  is  capable  of 
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conducting  warfare,  the  OA/OMIES  employs  cases  as  templates  in  order  to  detect  whether  an 
operator  is  conducting  the  correct  course  of  action.  This  application  of  case-based  reasoning 
makes  the  task  of  the  OA/OMIES  tractable,  circumventing  the  need  to  implement  an  entire 
automated  operator,  while  still  providing  applicable  and  useful  aid  to  the  operator. 

Retrieving  a  case  that  is  suitably  similar  to  the  current  situation  in  a  simulation,  or  in  a 
real  mission,  will  reveal  the  set  of  principles  pertaining  to  that  situation  and  the  set  of  actions  that 
are  appropriate.  Cases  are  represented  in  a  constraint-based  fashion,  with  various  levels  of 
generality.  One  case  could  represent  littoral  ASW  in  general,  suggesting  the  applicability  of 
principles  of  shallow-water  physics,  constraints  of  enemy  maneuvering,  appropriate  search 
techniques,  etc.  Another  case  could  represent  a  particular  localization  situation  in  littoral  ASW, 
suggesting  also  sonobuoy  placement  pattern  and  depth  settings,  etc. 

The  utility  of  this  case-based  approach  is  enhanced  by  the  ability  of  experts  to  expand  the 
case  base  by  simulating  further  situations,  and  then  entering  them  as  cases  associated  with  certain 
principles;  that  case  would  then  be  usable  immediately  in  the  OA/OMIES.  In  this  sense,  the 
OA/OMIES  forms  the  larger  part  of  an  authorable  intelligent  tutoring  system.  As  the  case  base 
expands  to  include  specific  cases  for  a  variety  of  situations,  its  enhancement  of  the  SH-60R 
interface  will  become  more  specific  and  detailed.  Because  the  case  base  associates  the  principle 
hierarchy  with  certain  mission  situations,  the  two  collectively  form  the  underlying  contextual 
knowledge  base  of  the  OA/OMIES. 

Scenarios 

Cases  are  distinct  in  the  OA/OMIES  from  assessment  scenarios,  which  are  sets  of  initial 
conditions  of  simulations.  As  in  a  conventional  training  simulation  (which  could  be  used  for  this 
purpose),  each  scenario  presents  to  the  operator  a  set  of  precise  circumstances,  and  a  current 
mission  objective,  in  which  to  act.  The  interface  to  the  simulation  is  a  virtual  recreation  of  the 
operator's  station.  As  the  operator  attempts  to  execute  the  mission  and  events  unfold,  the 
OA/OMIES  retrieves  cases  from  the  case  base  which  match  the  particular  conditions  of  that 
moment  in  the  scenario.  The  resulting  cases  will  determine  the  appropriate  response  to  those 
situations,  and  when  the  operator  differs  from  these  actions,  the  OA/OMIES  will  assume  a 
deficiency  in  the  associated  principles  and  include  this  deficiency  in  the  mental  model.  Figure  2 
shows  the  design  of  the  OA/OMIES  assessment  module. 
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Figure  2.  OA/OMIES  Assessment  Module 


Further  execution  of  scenarios  allows  the  OA/OMIES  to  form  an  accurate  mental  model 
over  all  domains  represented  in  the  principle  hierarchy.  As  the  operator  continues  to  be  assessed, 
the  system  can  choose  subsequent  scenarios  containing  conditions  that  focus  on  certain  sets  of 
principles,  in  order  to  refine  the  mental  model  most  efficiently.  Because  the  complete  set  of 
principles  is  hierarchical,  the  OA/OMIES  can  also  determine  that  the  operator  has  a  broad 
deficiency  in  an  entire  subdomain,  mark  that  branch  of  the  hierarchy  as  not  well  known  by  the 
operator,  and  focus  on  other  principles  in  subsequent  assessment.  Of  course,  any  operator's 
mental  model  can  be  altered  by  future  assessment  sessions. 

The  OA/OMIES  can  also  identify  deficiencies  on  a  higher  operational  order  than  specific 
domain  knowledge,  such  as  the  inabilities  to  perform  differing  kinds  of  tasks  simultaneously,  to 
deal  with  an  overwhelming  amount  of  simultaneous  data,  and  to  maintain  situational  awareness 
by  proper  monitoring  of  different  sensor  displays.  These  principles  are  those  in  which  any 
operator  will  have  some  level  of  deficiency;  that  is,  thresholds  of  complexity  exist  above  which 
even  the  best  operator  will  be  unable  to  perform  effectively.  The  system  will  be  sensitive  to  these 
kinds  of  deficiencies  in  addition  to  those  concerning  operational  knowledge.  It  is  also  important 
to  note  that  some  principles  might  be  more  appropriately  tested  in  a  traditional  straightforward 
test,  because  their  effects  on  mission  simulations  are  difficult  to  isolate.  Device  principles  are  an 
example,  including  knowledge  about  how  to  physically  operate  sensors  and  the  console  itself.  An 
operator's  understanding  of  these  principles  can  be  tested  by  the  OA/OMIES  through  a  standard 
test  in  addition  to  simulation-based  assessment. 

Enhancement 

The  enhancement  that  the  OA/OMIES  provides  on  board  the  SH-60R  is  a  continuously 
repeating  process  consisting  of  two  primary  stages:  identification  and  display.  The  identification 
stage  is  similar  in  nature  to  the  assessment  module,  in  that  it  consists  of  determining  relevant 
principles  by  retrieval  of  cases  similar  to  the  current  mission  situation.  The  system  has  a  mental 
model  of  the  operator  available  to  it,  which  describes  that  operator's  deficiencies  in  certain 
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domains,  which  will  in  turn  suggest  the  interface  enhancement  which  will  best  facilitate  the 
operator  at  that  moment.  Determination  of  the  most  appropriate  enhancement  can  be  expressed  as 
the  following  process: 


(context,  principles,  deficiencies)  =>  enhancement 

Context  consists  of  the  current  mission  situation  (e.g.,  ASW  tracking  of  a  particular  kind 
of  sub),  available  sensor  data  (e.g.,  known  contacts,  tracks,  speed/heading  data),  and  the  activities 
of  the  operator  (e.g.,  sensor  and  console  display  settings).  Context  is  used  differently  in  each 
phase  of  enhancement:  generally,  in  order  to  retrieve  the  most  similar  case  and  identify  the 
relevant  principles  on  which  to  act,  and  specifically,  when  determining  the  enhancement  to 
produce.  The  first  sense  is  general  in  that  a  retrieved  case  is  not  necessarily  identical  to  the 
current  situation,  but  is  sufficiently  similar  to  identify  the  relevant  principles  to  be  employed  in 
the  enhancement  process. 

Identification 

The  case-based  retrieval  facility  allows  for  rapid  real-time  identification  of  the  most 
similar  case  in  the  case  base,  which  has  associated  with  it  a  set  of  relevant  operational  principles. 
The  principles  identified  in  this  fashion  are  compared  to  the  mental  model  of  the  operator,  to 
determine  which  of  them  the  operator  may  be  lacking.  Should  a  deficiency  arise,  the  OA/OMIES 
will  execute  the  interface  aid  associated  with  the  principle  in  question.  In  the  presence  of 
multiple  deficiencies,  the  system  will  determine  the  most  appropriate  type  of  aid  as  a  function 
(function  F  below)  of  the  principle's  criticality,  the  extent  of  the  operator's  deficiency,  the 
immediacy  of  the  context,  and  the  effectiveness  of  the  enhancement  in  terms  of  speed  and 
specificity.  The  identification  phase  of  enhancement  can  be  expressed  as  the  following 
breakdown  of  the  general  process: 


contextgeneral  =>  case 
case  =>  relevant  principles 
relevant  principles,  mental  model  =>  relevant  aids 
F(relevant  aids,  relevant  principles,  mental  model,  contextgeneral)  =>  aid 


Display 

The  aid  identified  can  be  of  various  forms:  an  alert  or  help  text,  a  specialized  application 
such  as  a  classification  aid,  a  display  alternate  to  that  shown  on  the  operator's  console  with 
annotation  or  declutter,  or  a  display  of  data  from  a  different  sensor  altogether.  The  aid  can  also 
have  varying  content:  general  help  detailing  proper  equipment  operation  or  relevant  tactics, 
advice  consisting  of  suggested  configurations  or  actions  that  should  (or  should  not)  be  taken, 
information  or  appropriate  conclusions  of  which  the  operator  might  not  be  aware,  or  specific 
facilitation  of  particular  tasks  such  as  a  case-based  acoustic  signature  classification  aid.  Context- 
insensitive  aids  are  simply  shown  on  the  screen;  context-sensitive  aids,  such  as  suggestions  for 
sonobuoy  settings  and  placement  patterns,  refers  to  the  current  scenario  data  in  order  to  produce 
specific  advice.  As  opposed  to  the  identification  phase,  here  the  OA/OMIES  makes  use  of 
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specific  quantitative  context  information,  as  required  by  the  aid;  for  example,  a  buoy  placement 
aid  can  refer  to  current  tracking  data  as  well  as  knowledge  about  the  enemy  submarine,  where  a 
classification  aid  can  draw  from  real-time  acoustic  data.  The  final  step  of  enhancement,  then,  is 
the  following: 

aid,  contextspecific  =>  enhancement 


Sensor  Operator's  Associate 

The  physical  integration  of  the  OA/OMIES  with  the  SH-60R  platform  is  as  an  auxiliary 
display,  probably  a  laptop  beside  the  console  on  which  enhancements,  when  appropriate,  will 
appear.  As  an  associate  entity,  the  system  will  be  noncritical  to  the  operator's  console  operation 
and  mission  completion,  and  its  failure  would  incur  no  detriment  to  the  operator's  usual 
activities.  The  display  will  usually  be  blank,  to  be  unobtrusive  and  nondistracting  to  the  operator 
when  not  needed,  and  so  that  the  operator  does  not  get  used  to  ignoring  it  or  become  dependent 
upon  it  for  help.  With  an  expert  operator,  no  enhancement  may  ever  be  activated.  While  the 
OA/OMIES  that  has  been  designed  is  suitable  for  both  ATO  and  SO,  the  Phase  I  prototype  will 
be  implemented  as  an  SO's  associate,  and  orient  on  the  tasks  that  the  SO  must  perform  in  mission 
execution. 

Other  uses  of  the  OA/OMIES 

While  the  OA/OMIES  as  described  could  be  used  directly  for  specialized  just-in-time 
training,  and  for  post-mission  debriefing,  the  assessment  module  of  the  OA/OMIES  could  easily 
be  built  into  a  full-fledged  intelligent  tutoring  system,  with  most  of  the  work  already  having  been 
done.  SHAI  has  extensive  experience  in  case-based,  simulation-oriented  intelligent  tutoring 
systems,  using  the  approach  to  assessment  described  in  this  design.  The  case-based  approach 
offers  easier  and  more  intuitive  alternatives  to  the  difficult  and  time-consuming  processes  of 
knowledge  elicitation  and  training  course  authoring,  and  also  allows  for  extensibility  of  the 
knowledge  base  without  reimplementation.  It  is  also  important  to  note  that  the  information 
gathered  by  the  system  could  also  be  useful  in  the  interface  design  process  itself.  Problems  or 
mistakes  that  most  operators  experience  might  indicate  a  fault  or  inefficiency  in  the  interface,  and 
effort  can  be  exerted  to  correct  that  problem  rather  than  to  train  operators  in  unwieldy  tasks. 
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6.0  Phase  I  Prototype 

The  Phase  I  prototype  provides  absolute  proof  of  the  feasibility  of  our  ideas.  It  was 
developed  in  a  two-month  time  period,  from  scratch  in  the  Kappa  rapid  prototyping  environment. 
The  Phase  I  prototype  was  kept  unclassified  by  not  using  the  correct  tactics,  though  their  form  is 
preserved.  For  demonstration  purposes,  many  of  the  decisions  made  by  the  ATO  and  SO  are 
combined. 


6.1  Functionality 

The  Phase  I  prototype  implements  all  phases  of  the  full-scale  OM/OMIES,  though  on  a 
very  narrow  part  of  the  SH-60R  domain.  It  includes  both  an  assessment  module  which  tests 
operator  knowledge  in  scenarios  running  in  a  tactical  simulation,  and,  an  enhancement  module. 
The  assessment  module  assembles  an  operator  model,  consisting  of  what  Principles  the  operator 
is  weak  and  strong  in.  The  assessment  also  performs  assessment  efficiently.  That  is,  if  it 
determines  that  he  knows  very  little  or  very  much  about  one  part  of  the  SH-60R  domain,  it  marks 
the  entire  area  accordingly  and  moves  on  to  scenarios  covering  other  areas. 

The  enhancement  system  uses  the  operator  model,  in  the  context  of  the  current  situation, 
to  provide  the  appropriate  enhancements.  Enhancements  included  in  the  Phase  I  prototype 
include  recreation/improvement  of  existing  sensor  displays,  knowledge-based  advice,  advisories, 
warnings,  suggestions,  explanations,  and  domain  information  (both  general  and  tailored  to  the 
context). 

6.2  Phase  I  Design 

To  a  large  degree,  the  design  of  the  Phase  I  prototype  follows  the  Phase  II  design  given  in 
Section  5.2.  That  general  design  is  not  repeated  here,  but  only  the  aspects  that  are  particular  to 
the  Phase  I  prototype.  Figure  3  shows  the  high-level  contents  of  the  prototype. 
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Figure  3.  OA/OMIES  Prototype 


It  contains  a  simulation  that  supports  ASW/ASuW  scenarios,  a  monitor  that  builds  mental 
models  in  assessment  mode  and  provides  adaptations  in  enhancement  mode,  a  hierarchy  of 
principles  representing  the  operation  knowledge  of  interest,  and  set  of  cases  that  is  used  in  case- 
based  reasoning  to  determine  the  applicability  to  mission  situations,  and  scenarios  in  which  to 
carry  out  assessment  and  enhancement.  Figure  4  shows  the  operational  architecture  of  the 
prototype,  and  is  followed  by  descriptions  of  the  significant  components. 
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Figure  4.  OA/OMIES  Prototype  Operational  Architecture 
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Phase  I  Components 

1.  Simulation 

2.  Control 

3.  Human  Interface 

4.  Assessment 

5.  Enhancement 


1.  Simulation 

The  simulation  component  of  the  prototype  provides  the  platform  for  operator  assessment  and 
interface  enhancement.  It  plays  the  roles  of  both  training  simulation,  in  which  the  system 
assesses  the  operator’s  expertise  during  simulated  missions,  and  the  SH-60R  platform  itself,  in 
demonstrating  some  of  the  interface  enhancements  that  would  be  made  available  during  a 
mission.  The  simulation  models  the  interaction  of  physical  objects  in  a  tactical  ASW/ASuW 
domain,  and  can  include  various  types  of  submarines  and  ships,  as  well  as  the  SH-60R,  weapons 
available  to  various  platforms,  and  miscellaneous  entities  such  as  decoys.  The  uppermost  portion 
of  Figure  5  shows  the  set  of  such  objects  (‘PhysObj’s)  that  are  represented. 
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Figure  5.  Simulation  and  Control  Entities  in  the  OA/OMIES  Prototype 

These  platforms  can  have  onboard  any  of  the  other  major  set  of  objects  in  the  simulation,  that  of 
sensors.  Sensor  objects  collect  data  from  the  simulation  and  provide  that  data  for  use  by  their 
platforms.  In  the  scenarios  included  in  this  prototype,  submarines  and  ships  are  equipped  only 
with  simple  sonar  sensors,  to  allow  them  the  information  they  need  for  evasion  and  attack,  while 
the  SH-60R  has  onboard  its  full  suite  of  sensors,  each  of  which  is  constantly  generating  data  and 
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making  it  available  to  the  operator  while  the  simulation  is  running,  to  produce  a  simplified 
replication  of  an  actual  ATO  station.  The  set  of  sensors  implemented  in  this  prototype  is  shown 
in  Figure  5. 

2.  Control 

While  the  simulation  component  models  the  physical  environment  and  the  objects  in  it,  updating 
their  states  over  time,  as  well  as  providing  data  for  sensors  being  used  in  the  simulation,  the 
objects  require  control  entities  to  perform  actions.  The  OA/OMIES  prototype  includes  an  agent 
architecture  that  can  be  used  to  simulate  the  commanders  of  various  platforms,  which  are  capable 
of  analyzing  the  environment  via  sensor  data  and  performing  actions  toward  the  completion  of  a 
particular  mission.  Each  scenario  included  in  this  prototype  specifies  a  set  of  ships  and 
submarines  that  are  present,  as  well  as  a  mission  statement  for  each,  which  will  be  used  by  the 
agents  associated  with  each  object  in  making  decisions  while  the  simulation  is  being  run.  The 
types  of  agents  are  shown  in  Figure  5. 

The  agents,  as  well  as  the  human  operator  using  the  prototype,  manipulate  the  simulation  by 
assigning  tasks  to  the  object  with  which  they  are  associated.  These  tasks  are  shown  in  Figure  5, 
and  include  the  deployment  of  weapons,  sensors,  or  decoys,  transit,  pursuit,  and  the  like.  The 
SubCommander  agent,  for  example,  may  notice  a  contact  on  the  sonar  aboard  his  submarine,  and 
attempt  to  evade  by  deploying  a  decoy  and  diving  below  the  layer,  using  the  DeployDecoy  and 
GoToPosition  tasks. 
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3.  Human  Interface 

The  SH-60R  through  the  same  set  of  possible  tasks,  but  instead  of  being  provided  by  an  agent 
control  entity,  these  tasks  result  from  the  human  operator’s  use  of  the  OA/OMIES  interface, 
shown  in  Figure  6. 


Hook  Mode 
(?  FTP 
r  Mark 
C  DipPoint 
C  BuoyDropPoint 
C  DeployTorpedo 
C  AirPlan 
C  SetCenter 
C  RangeBearingLine 


loy  Type 


FAR  m 


py  Depth 


B 


Sensors 

r  FLIR 

TMMR 

r  IFF 

r  MAD 
TALFS 


MMR  Mode 
(*  Normal 
C  LPD 


ai 


ALFS  Depth 


400 


0  feet 


Action 


Halt 


Analysis 
ALI  Time 


a 


TACT  | 

Fliaht  Display 
Position 

1  fur  1 

X=0.00 

Y=0.00 

|  radar] 

Course 

|  Ooo  | 

|  IFF  | 

|  ESM  ! 

Speed  (Ms) 

|  (TiJoo  1 

Time 

ACSTj 

|  OVLY  j 

|  00:00:00  | 

Line 

Range  (nm) 

1  ^1 

1  -  1 

Bearing 

|  TogEnhj 

I - - - 1 

|:w| 

Environment 
MDR  [nm)  i 

View  | 

|  2^500  | 

OMIA  Modi 

TSR  (nm) 

Asstssmott  1 

|  5.000  | 

— - - 1 

Mission  Plust 

LayerDepth  (ft) 

START  [ 

L  150  ...1 

Simulator 

Tim 

Alii"' 

1 

e  Scale 

- ±1 

1  60 

1  X 

|  /  Reset 

Hiil! 

|  Start 

0.117  nm 

Figure  6.  OA/OMIES  Operator  Interface 


The  prototype  interface  represents  a  simplification  of  the  SH-60R  human-machine  interface,  in 
addition  to  some  controls  to  allow  the  operator  to  control  the  helicopter  and  the  simulation  itself. 
The  primary  display  (upper  left)  shows  the  sensor  data  or  tactical  information  corresponding  to 
the  display  mode  selected  (upper  center,  left),  including  modes  for  each  sensor  and  for  overlays 
of  more  than  one.  Various  flight  information  is  also  provided  (upper  center,  right).  The  operator 
controls  the  SH-60R  via  the  lower  section  of  windows,  which  allow  the  establishment  of  FTPs, 
drop  points,  dip  points,  weapon  deployment  points,  airplans,  and  the  like,  which  are  executed 
automatically  by  the  SH-60R  ‘pilot’  (lower  left).  A  panel  for  the  deployment  of  various  sensors 
is  adjacent,  as  well  as  a  control  for  the  selection  of  ALI  integration  time.  Various  other  controls 
allow  stopping,  starting,  and  time  compression  of  the  simulation,  as  well  as  zooming  and 
centering  the  primary  display.  The  two  rightmost  windows  shown  are  reserved  for  enhancements, 
as  well  as  messages  that  annotate  the  demonstration  sequence. 
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4.  Assessment 

The  assessment  module  of  the  OA/OMIES  prototype  follows  the  design  detailed  in  section  5.2. 
The  principle  hierarchy  used  as  a  basis  for  the  generation  of  operator  mental  models  is  shown  in 
Figure  7. 
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Figure  7.  OA/OMIES  Prototype  Principle  Hierarchy 


The  incomplete  and  simplified  hierarchy  used  in  the  demonstration  prototype  represents,  at  the 
highest  level,  various  types  of  knowledge  relevant  to  the  scenarios  used  in  the  prototype: 
physical  principles  (P_Physics)  governing  the  ocean  environment  and  affecting  factors  such  as 
the  propagation  of  sound,  mission  completion  principles  (P_Mission)  including  methods  for 
buoy  placement  and  appropriate  tactical  use  of  sensors,  equipment  principles  (P_Equipment)  of 
operation  for  sensors  and  weapons,  and  enemy  intelligence  principles  (P_Enemy)  of  enemy 
capabilites  and  of  enemy  behavior  in  certain  situations.  All  of  the  principles  associated  with  the 
scenarios  used  in  the  prototype  lie  at  the  bottom  level  inside  these  groups.  Omitting  the 
intermediary  principles,  brief  descriptions  are  as  follows  (note  that  not  every  principle  is 
demonstrated  explicitly  in  our  demo  sequence): 

P_OceanLayer:  how  the  ocean  layer  affects  sound  propogation,  and  how  to  place  hydrophones 
to  exploit/avoid  them 

P_Correlation:  correlation  of  contacts  on  different  sensors  to  improve  confidence, 
classification,  etc. 

P_ActiveSensing:  the  effect  of  active  sensing  on  covertness 

P_RADAR_LPD:  the  use  of  the  LPD  mode  of  the  MMR  for  covertness 

P_BPSpacing:  the  optimal  spacing  of  passive  sonobouys  in  patterns 

P_BPLOB:  the  appropriate  passive  sonobuoy  pattern  to  investigate  a  line  of  bearing 

P_TRITACExp:  TRITAC  expansion  of  split  buoys 

P_DITACExp:  DITAC  expansion  of  hot  buoys 
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P_AnticipateSub:  keeping  sensor  deployment  ahead  of  expected  sub  heading 
P_AttackCriteria:  criteria  for  attack  using  various  sensor  data 
P_ALI:  appropraite  use  of  ALI  time  constant  for  various  mission  phases 
P_BackNoise:  the  effect  of  background  noise  on  acoustic  sensors 
P_Decoys:  the  characteristics  of  enemy  decoys,  and  the  use  thereof 
P_EnemyDepth:  expected  depths  of  enemy  subs,  given  certain  supected  intent 
P_EnemySpeed:  expected  speeds  of  enemy  subs,  given  certain  supected  intent 

5.  Enhancement 

The  enhancements  demonstrated  by  the  OA/OMIES  prototype  are  of  two  basic  types:  graphic 
and  textual.  Each  provides  a  suggestion  to  the  operator,  based  on  the  current  environmental  data 
and  tactical  situation,  and  also  provides  an  explanation  of  the  principle  or  principles  motivating 
the  enhancement.  For  example,  an  enhancement  that  provides  a  recommended  hydrophone  depth 
for  a  passive  sonobouy  consults  and  presents  the  known  ocean  layer  depth  and  relevant 
intelligence  about  the  enemy  operating  depth,  based  on  platform  type  and  suspected  intent. 


6.3  Demonstration  Sequence  Overview 

The  most  important  concept  of  the  OA/OMIES  is  that  it  adapts  itself  differently  to 
different  operators.  To  show  this  capability  requires  that  the  demonstration  sequence  include 
sub-sequences  for  two  different  operators.  The  first  sub-sequence.  Operator  1,  starts  with  the 
Passive  Line  of  Bearing  (LOB)  scenario.  Operator  1  performs  very  poorly  and  the  system 
terminates  the  scenario  when  it  becomes  obvious  that  Operator  1  has  a  complete  lack  of 
knowledge  in  this  area.  It  then  picks  another  scenario,  to  test  knowledge,  not  tested  in  the 
previous  scenario.  This  scenario  is  a  Passive  Datum  Assessment  scenario.  Operator  1  performs 
most  of  the  actions  correctly,  showing  he  is  knowledgeable  in  this  area  and  can  apply  the 
relevant  tactical  principles.  Operator  1  is  then  forced  to  perform  a  mission  scenario.  For 
demonstration  purposes,  the  simulation  plays  the  role  of  the  actual  SH-60R  cockpit  and 
environment.  During  the  mission  scenario,  the  Operator  receives  enhancements  appropriate  for 
the  principles  that  the  has  demonstrated  a  weakness  in. 

Operator  2  performs  the  second  sub-sequence.  He  will  make  different  mistakes  than 
Operator  1,  reflecting  differing  knowledge.  This  will  cause  him  to  get  an  almost  completely 
different  set  of  enhancements.  He  starts  with  the  same  initial  scenario  (Passive  LOB).  He 
performs  much  better  than  Operator  1  and  achieves  detection  (which  Operator  1  was  never  able 
to  accomplish).  He  the  then  moves  into  localization,  tracking,  and  attack.  He  makes  some 
mistakes  but  is  still  able  to  accomplish  a  kill.  Because  he  was  tested  on  the  principles  relating  to 
these  tasks,  unlike  Operator  1 ,  he  is  able  to  skip  the  second  assessment  scenario.  He  is  then 
forced  to  perform  the  same  mission  scenario  as  Operator  1 ,  but  receives  a  very  different  set  of 
enhancements  during  this  same  mission,  because  the  state  of  his  knowledge  is  so  different  than 
A.  Much  more  detail  is  given  on  the  demonstration  sequence  in  Appendix  A,  including  a  large 
number  of  screen  dumps. 
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7.0  Future  Work 

The  ultimate  goal  of  this  project  is  a  fielded,  operational  system  which  performs  off-line 
assessment  and  on-line  OMI  enhancment,  on-board  the  SH-60R,  for  both  the  ATO  and  SO 
positions.  This  is  an  emormous  scope  which  must  be  scaled-back  and  prioritized  for  Phase  II.  In 
Phase  II,  we  would  produce  an  operational  prototype,  ready  for  testing  and  evaluation,  probably 
interfaced  through  an  RS-232  port  to  a  land-based  functional  cockpit  mock-up.  The  Phase  II 
system  would  handle  a  subset  of  the  applicable  knowledge  and  tasks  of  the  SO  or  ATO. 

The  ultimate  system,  in  addition  to  interfacing  to  the  actual  SH-60R  avionics  must  also 
interface  to  an  SH-60R  trainer,  for  OA/OMIES  testing,  off-line  assessment,  and  for  training  the 
crew  in  the  use  of  the  on-line  enhancement. 

Future  work  will  include  both  the  development  of  applicable  OMI  enhancments  by  SHAI 
as  well  as  incorporation  of  enhancments  developed  by  others.  The  Decision  Support  System 
(DSS)  is  one  example.  Our  architecture  minimizes  the  difficulty  of  incorporating  enhancments 
developed  by  other  organizations.  SHAI  is  qualified  to  develop  several  different  types  of 
enhancments.  Which  ones  we  develop,  and  which  ones  will  be  developed  by  others,  must  be 
decided. 

The  OA/OMIES  will  useful  during  mission  planning,  specifically  for  mission  rehersal. 
The  crew  can  be  put  through  several  scenarios  similar  to  ones  that  are  expected  in  the  course  of 
the  real  missions.  The  system  can  evaluate  the  crew’s  responses  and  determine  what  elements 
they  are  weak  in.  The  relevent  enhancments  will  thus  be  primed  for  the  real  mission,  and,  for 
additional  “dry  runs”,  if  desired. 

Another  obvious  extension  to  the  OA/OMIAS  is  an  Intelligent  Tutoring  System  (ITS). 

The  most  difficult  aspects  of  ITSs  are  generally  assement  and  operator  model  building.  Since 
these  are  already  being  accomplished  by  the  OA/OMAS,  only  sttraight-forward  and  minor 
additions  are  required  to  give  it  an  ITS  capability.  Even  simply  tying  the  proinciples  back  to  an 
electronic  version  (preferrably  multi-media)  of  the  tactical  manuals,  would  go  a  long  way  in  this 
direction.  Since  the  OA/OMIES  develops  lists  of  poorly  applied  principles,  an  ITS  could 
retrieve  and  present  the  parts  of  the  tactical  manuals  specifical  relevant  ot  a  particlar  operators 
deficienceis.  Another  straight-forward  addition  for  ITS  use  would  be  the  ability  to  retrieve 
scenarios  for  a  particular  operator  based  on  his  particular  deficiencies.  The  could  be  used  as 
scenario  exercises  for  practice  or  presented  as  examples  for  teaching.  Finally,  attaching  an 
expert’s  explanations  to  the  expected  operator  actions  would  provide  a  debriefing  capability. 
When  combined  with  the  mission  rehersal  capability,  the  ITS  provides  just-in-time-training. 

There  are  different  types  of  evaluation  functions  which  the  OA/OMIES,  by  its  nature,  can 
easily  support.  The  first  is  operator  performance  evaluation.  The  OA/OMIES  is  assessing  the 
operator  in  operational  scenarios  and  building  a  corresponding  model  of  the  state  of  his 
knowledge.  This  model  can  be  output  in  human-readable  form  and  used  directly  for  evaluating 
the  operator’s  performance  of  neccessary  tasks  and  his  related  knowledge.  Second,  the 
OA/OMIES  can  be  used  to  examine  how  intuitive  the  OMI  is  and  how  effective  training  is.  By 
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examining  the  information  across  different  operators,  commonalitites  can  be  found.  If  operators 
frequently  have  difficulty  with  the  same  aspects,  this  may  indicate  either  a  deficiency  in  training, 
or  even  in  the  OMI  itself. 


Finally,  during  the  development  of  the  OA/OMIES,  SHAI  will  be  developing  the 
knowledge  strucutres  for  the  domain,  which  could  bge  used  to  suport  additional  purposes.  These 
include  ITSs  and  particular  OMI  enhancments,  especially  in  the  area  of  ATO  and  SO  Associates. 
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